{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "01_neural_network_regression_in_tensorflow_interactive.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UPgo18-N1gSi"
      },
      "source": [
        "# 01. Neural Network Regression with TensorFlow (INTERACTIVE VERSION)\n",
        "\n",
        "There are many definitions for a [regression problem](https://en.wikipedia.org/wiki/Regression_analysis) but in our case, we're going to simplify it to be: predicting a number.\n",
        "\n",
        "For example, you might want to:\n",
        "- Predict the selling price of houses given information about them (such as number of rooms, size, number of bathrooms).\n",
        "- Predict the coordinates of a bounding box of an item in an image.\n",
        "- Predict the cost of medical insurance for an individual given their demographics (age, sex, gender, race).\n",
        "\n",
        "In this notebook, we're going to set the foundations for how you can take a sample of inputs (this is your data), build a neural network to discover patterns in those inputs and then make a prediction (in the form of a number) based on those inputs.\n",
        "\n",
        "## What we're going to cover\n",
        "\n",
        "Specifically, we're going to go through doing the following with TensorFlow:\n",
        "- Architecture of a regression model\n",
        "- Input shapes and output shapes\n",
        "  - `X`: features/data (inputs)\n",
        "  - `y`: labels (outputs)\n",
        "- Creating custom data to view and fit\n",
        "- Steps in modelling\n",
        "  - Creating a model\n",
        "  - Compiling a model\n",
        "    - Defining a loss function\n",
        "    - Setting up an optimizer\n",
        "    - Creating evaluation metrics\n",
        "  - Fitting a model (getting it to find patterns in our data)\n",
        "- Evaluating a model\n",
        "  - Visualizng the model (\"visualize, visualize, visualize\")\n",
        "  - Looking at training curves\n",
        "  - Compare predictions to ground truth (using our evaluation metrics)\n",
        "- Saving a model (so we can use it later)\n",
        "- Loading a model\n",
        "\n",
        "Don't worry if none of these make sense now, we're going to go through each.\n",
        "\n",
        "## How you can use this notebook\n",
        "\n",
        "Run each line of code, and do the interactive quizzes which will be marked with this ❓ emoji."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Preparing Interactive Environment\n",
        "\n",
        "This notebook uses pythonwhat package to add coding challenges to the notebook kind of like Kaggle Learn Course Notebooks"
      ],
      "metadata": {
        "id": "44w_D8neP5FV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install pythonwhat"
      ],
      "metadata": {
        "id": "fKWYlo3fhI3j"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from IPython.core.magic import register_cell_magic\n",
        "@register_cell_magic\n",
        "def run_and_save(line, cell):\n",
        "    'Run and save python code block to a file'\n",
        "    with open(line, 'wt') as fd:\n",
        "        fd.write(cell)\n",
        "    code = compile(cell, line, 'exec')\n",
        "    exec(code, globals())"
      ],
      "metadata": {
        "id": "alwJhPdH2JrX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# make all checking functions available\n",
        "from pythonwhat.test_exercise import prep_context\n",
        "_, ctxt = prep_context()\n",
        "globals().update(ctxt)\n",
        "from pythonwhat.test_exercise import setup_state"
      ],
      "metadata": {
        "id": "9fglCmPChpcu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from pathlib import Path\n",
        "\n",
        "class question:\n",
        "  def __init__(self, sol_code, hint='', filename='student_input.py', pre_code=''):\n",
        "    self._pre_code = pre_code\n",
        "    self._sol_code = sol_code\n",
        "    self._hint = hint\n",
        "    self._filename = filename\n",
        "\n",
        "  def submit_answer(self):\n",
        "    stu_code = Path(self._filename).read_text()\n",
        "    setup_state(stu_code=stu_code,\n",
        "                sol_code=self._sol_code,\n",
        "                pec=self._pre_code)\n",
        "    \n",
        "  def hint(self):\n",
        "    print(self._hint)\n",
        "\n",
        "  def solution(self):\n",
        "    print(self._sol_code)"
      ],
      "metadata": {
        "id": "ttCCIrwoin2x"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### ❔ Instantiate Questions"
      ],
      "metadata": {
        "id": "yTJtNFql0VZQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# TODO - figure out how to indent multiple line quotes\n",
        "\n",
        "q_1 = question(sol_code='single_example = X[0]',\n",
        "               hint='Use [] brackets',\n",
        "               pre_code='import tensorflow as tf\\nX = tf.constant([-7.0, -4.0, -1.0, 2.0, 5.0, 8.0, 11.0, 14.0])'\n",
        "              )\n",
        "q_2 = question(sol_code=\"\"\"input_shape = X[0].shape\n",
        "output_shape = y[0].shape\"\"\",\n",
        "               hint='Use .shape',\n",
        "               pre_code=\"\"\"import tensorflow as tf\n",
        "X = tf.constant([-7.0, -4.0, -1.0, 2.0, 5.0, 8.0, 11.0, 14.0])\n",
        "single_example = X[0]\n",
        "y = tf.constant([3.0, 6.0, 9.0, 12.0, 15.0, 18.0, 21.0, 24.0])\n",
        "\"\"\")\n",
        "\n",
        "q_3 = question(sol_code=\"\"\"model = tf.keras.Sequential([\n",
        "  tf.keras.layers.Dense(1)\n",
        "])\n",
        "\n",
        "# Compile the model\n",
        "model.compile(loss=tf.keras.losses.mae, # mae is short for mean absolute error\n",
        "              optimizer=tf.keras.optimizers.SGD(), # SGD is short for stochastic gradient descent\n",
        "              metrics=[\"mae\"])\n",
        "\n",
        "layer_name = model.layers[0].name[0:5]\n",
        "units = model.layers[0].units\n",
        "loss = model.loss\n",
        "optimizer = model.optimizer.get_config()['name']\n",
        "\"\"\",\n",
        "              hint='See Improving Model Section',\n",
        "              pre_code=\"\"\"import tensorflow as tf\n",
        "tf.random.set_seed(42)\n",
        "              \"\"\")\n",
        "\n",
        "q_4 = question(sol_code='model.predict([17.0])')\n",
        "\n",
        "q_5 = question(sol_code='model.fit(X_train, y_train, epochs=100, verbose=0)')\n",
        "\n",
        "q_6 = question(sol_code='model.summary()')\n",
        "\n",
        "q_7 = question(sol_code=\"\"\"mse = tf.metrics.mean_squared_error(y_true=y_test,\n",
        "y_pred=y_preds.squeeze())\n",
        "\"\"\",\n",
        "hint='Look up tf.metrics mean squared error on google')\n",
        "\n",
        "q_8 = question(sol_code=\"\"\"tf.keras.layers.Dense(1),\n",
        "tf.keras.layers.Dense(1)\"\"\")\n",
        "\n",
        "q_9 = question(sol_code=\"\"\"model_2.predict(X_test)\n",
        "\"\"\")\n",
        "\n",
        "q_10 = question(sol_code=\"\"\"mae_3 = mae(y_test, y_preds_3.squeeze()).numpy()\n",
        "mse_3 = mse(y_test, y_preds_3.squeeze()).numpy()\n",
        "\"\"\",\n",
        "pre_code=\"\"\"\n",
        "import tensorflow as tf\n",
        "import numpy as np\n",
        "\n",
        "def mae(y_test, y_pred):\n",
        "  return tf.metrics.mean_absolute_error(y_test,y_pred)\n",
        "  \n",
        "def mse(y_test, y_pred):\n",
        "  return tf.metrics.mean_squared_error(y_test,y_pred)\n",
        "\n",
        "y = np.arange(-90, 110, 4)\n",
        "y_test = y[40:]\n",
        "\n",
        "y_preds_3 = np.array([[15.879008],\n",
        "       [16.641926],\n",
        "       [17.404842],\n",
        "       [18.167759],\n",
        "       [18.930676],\n",
        "       [19.693592],\n",
        "       [20.456509],\n",
        "       [21.219427],\n",
        "       [21.982342],\n",
        "       [22.74526 ]], dtype=np.float32)\n",
        "\"\"\",\n",
        "hint='Use the mae() and mse() functions we built earlier and use .numpy()'\n",
        ")\n",
        "\n",
        "q_11 = question(sol_code=\"\"\"# Create a new model (same as model_2)\n",
        "insurance_model = tf.keras.Sequential([\n",
        "  tf.keras.layers.Dense(1),\n",
        "  tf.keras.layers.Dense(1)\n",
        "])\n",
        "\n",
        "# Compile the model\n",
        "insurance_model.compile(loss=tf.keras.losses.mae,\n",
        "                        optimizer=tf.keras.optimizers.SGD(),\n",
        "                        metrics=['mae'])\n",
        "\n",
        "# Fit the model\n",
        "insurance_model.fit(X_train, y_train, epochs=100)\n",
        "\"\"\")\n",
        "\n",
        "q_12 = question(sol_code='X_train_normal.shape, X_train.shape')"
      ],
      "metadata": {
        "id": "warQtYVV9A8m"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "etAu7oCZ8r_G"
      },
      "source": [
        "## Typical architecture of a regression neural network\n",
        "\n",
        "The word *typical* is on purpose.\n",
        "\n",
        "Why?\n",
        "\n",
        "Because there are many different ways (actually, there's almost an infinite number of ways) to write neural networks.\n",
        "\n",
        "But the following is a generic setup for ingesting a collection of numbers, finding patterns in them and then outputing some kind of target number.\n",
        "\n",
        "Yes, the previous sentence is vague but we'll see this in action shortly.\n",
        "\n",
        "| **Hyperparameter** | **Typical value** |\n",
        "| --- | --- |\n",
        "| Input layer shape | Same shape as number of features (e.g. 3 for # bedrooms, # bathrooms, # car spaces in housing price prediction) |\n",
        "| Hidden layer(s) | Problem specific, minimum = 1, maximum = unlimited |\n",
        "| Neurons per hidden layer | Problem specific, generally 10 to 100 |\n",
        "| Output layer shape | Same shape as desired prediction shape (e.g. 1 for house price) |\n",
        "| Hidden activation | Usually [ReLU](https://www.kaggle.com/dansbecker/rectified-linear-units-relu-in-deep-learning) (rectified linear unit) |\n",
        "| Output activation | None, ReLU, logistic/tanh |\n",
        "| Loss function | [MSE](https://en.wikipedia.org/wiki/Mean_squared_error) (mean square error) or [MAE](https://en.wikipedia.org/wiki/Mean_absolute_error) (mean absolute error)/Huber (combination of MAE/MSE) if outliers |\n",
        "| Optimizer | [SGD](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/SGD) (stochastic gradient descent), [Adam](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adam) |\n",
        "\n",
        "***Table 1:*** *Typical architecture of a regression network.* ***Source:*** *Adapted from page 293 of [Hands-On Machine Learning with Scikit-Learn, Keras & TensorFlow Book by Aurélien Géron](https://www.oreilly.com/library/view/hands-on-machine-learning/9781492032632/)*\n",
        "\n",
        "Again, if you're new to neural networks and deep learning in general, much of the above table won't make sense. But don't worry, we'll be getting hands-on with all of it soon.\n",
        "\n",
        "> 🔑 **Note:** A **hyperparameter** in machine learning is something a data analyst or developer can set themselves, where as a **parameter** usually describes something a model learns on its own (a value not explicitly set by an analyst).\n",
        "\n",
        "Okay, enough talk, let's get started writing code.\n",
        "\n",
        "To use TensorFlow, we'll import it as the common alias `tf` (short for TensorFlow)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FMqsqKpk7TrH"
      },
      "source": [
        "import tensorflow as tf\n",
        "print(tf.__version__) # check the version (should be 2.x+)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8clMYxrF6Mzv"
      },
      "source": [
        "## Creating data to view and fit\n",
        "\n",
        "Since we're working on a **regression problem** (predicting a number) let's create some linear data (a straight line) to model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G43tWFof6i7T"
      },
      "source": [
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Create features\n",
        "X = np.array([-7.0, -4.0, -1.0, 2.0, 5.0, 8.0, 11.0, 14.0])\n",
        "\n",
        "# Create labels\n",
        "y = np.array([3.0, 6.0, 9.0, 12.0, 15.0, 18.0, 21.0, 24.0])\n",
        "\n",
        "# Visualize it\n",
        "plt.scatter(X, y);"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9ONZF8un7_xy"
      },
      "source": [
        "Before we do any modelling, can you calculate the pattern between `X` and `y`?\n",
        "\n",
        "For example, say I asked you, based on this data what the `y` value would be if `X` was 17.0?\n",
        "\n",
        "Or how about if `X` was -10.0?\n",
        "\n",
        "This kind of pattern discover is the essence of what we'll be building neural networks to do for us."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zNCXxHnF6jjZ"
      },
      "source": [
        "## Regression input shapes and output shapes\n",
        "\n",
        "One of the most important concepts when working with neural networks are the input and output shapes.\n",
        "\n",
        "The **input shape** is the shape of your data that goes into the model.\n",
        "\n",
        "The **output shape** is the shape of your data you want to come out of your model.\n",
        "\n",
        "These will differ depending on the problem you're working on.\n",
        "\n",
        "Neural networks accept numbers and output numbers. These numbers are typically represented as tensors (or arrays).\n",
        "\n",
        "Before, we created data using NumPy arrays, but we could do the same with tensors."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XrXQ3m0prWXa"
      },
      "source": [
        "# Example input and output shapes of a regresson model\n",
        "house_info = tf.constant([\"bedroom\", \"bathroom\", \"garage\"])\n",
        "house_price = tf.constant([939700])\n",
        "house_info, house_price"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Yi3VWKH6sRrZ"
      },
      "source": [
        "house_info.shape"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kOxyr9sR6m9X"
      },
      "source": [
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Create features (using tensors)\n",
        "X = tf.constant([-7.0, -4.0, -1.0, 2.0, 5.0, 8.0, 11.0, 14.0])\n",
        "\n",
        "# Create labels (using tensors)\n",
        "y = tf.constant([3.0, 6.0, 9.0, 12.0, 15.0, 18.0, 21.0, 24.0])\n",
        "\n",
        "# Visualize it\n",
        "plt.scatter(X, y);"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zaPxco6E9i1_"
      },
      "source": [
        "Our goal here will be to use `X` to predict `y`.\n",
        "\n",
        "So our **input** will be `X` and our **output** will be `y`.\n",
        "\n",
        "Knowing this, what do you think our input and output shapes will be?\n",
        "\n",
        "#### ❓ Q1. Get a single example of X"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# This magic function saves your code for the code checker\n",
        "%%run_and_save student_input.py\n",
        "single_example = ____ # Your code here\n",
        "\n",
        "print(single_example)"
      ],
      "metadata": {
        "id": "hXPRQBovw3Gm"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j1oT1gmB9iX-"
      },
      "source": [
        "q_1.submit_answer()\n",
        "\n",
        "Ex().check_object('single_example').has_equal_value(incorrect_msg='Try Again')\n",
        "# If error message doesn't come up you are correct!\n",
        "print('Success!')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Lines below will give you a hint or solution code (uncomment the line and run it)\n",
        "# q_1.hint()\n",
        "# q_1.solution()"
      ],
      "metadata": {
        "id": "n3BM1gHEIEkC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### ❓ Q2. Get the input shape of a single example of X"
      ],
      "metadata": {
        "id": "M6OL-XJfJ86y"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%run_and_save student_input.py\n",
        "input_shape = ____ # YOUR CODE HERE\n",
        "\n",
        "# Take a single example of y and get its shape\n",
        "output_shape = ____ # YOUR CODE HERE\n",
        "\n",
        "print(input_shape)\n",
        "print(output_shape)"
      ],
      "metadata": {
        "id": "2LTTpFpWgzUd"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "q_2.submit_answer()\n",
        "\n",
        "Ex().check_object('input_shape').has_equal_value()\n",
        "Ex().check_object('output_shape').has_equal_value()\n",
        "print(\"Success!\") # this print statement will run assuming Ex() checker doesn't find any errors"
      ],
      "metadata": {
        "id": "2-Cr-3LRKki0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Lines below will give you a hint or solution code (uncomment the line and run it)\n",
        "# q_2.hint()\n",
        "# q_2.solution()"
      ],
      "metadata": {
        "id": "ymdJasQUKvBk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "input_shape, output_shape"
      ],
      "metadata": {
        "id": "76mEXnfBKhKB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m4txxFTL_EEX"
      },
      "source": [
        "Huh?\n",
        "\n",
        "From this it seems our inputs and outputs have no shape?\n",
        "\n",
        "How could that be?\n",
        "\n",
        "It's because no matter what kind of data we pass to our model, it's always going to take as input and return as ouput some kind of tensor.\n",
        "\n",
        "But in our case because of our dataset (only 2 small lists of numbers), we're looking at a special kind of tensor, more specificially a rank 0 tensor or a scalar."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dag5y4MPaTmc"
      },
      "source": [
        "# Let's take a look at the single examples invidually\n",
        "X[0], y[0]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fKtihU57_cOY"
      },
      "source": [
        "In our case, we're trying to build a model to predict the pattern between `X[0]` equalling `-7.0` and `y[0]` equalling `3.0`.\n",
        "\n",
        "So now we get our answer, we're trying to use 1 `X` value to predict 1 `y` value.\n",
        "\n",
        "You might be thinking, \"this seems pretty complicated for just predicting a straight line...\".\n",
        "\n",
        "And you'd be right.\n",
        "\n",
        "But the concepts we're covering here, the concepts of input and output shapes to a model are fundamental. \n",
        "\n",
        "In fact, they're probably two of the things you'll spend the most time on when you work with neural networks: **making sure your input and outputs are in the correct shape**.\n",
        "\n",
        "If it doesn't make sense now, we'll see plenty more examples later on (soon you'll notice the input and output shapes can be almost anything you can imagine).\n",
        "\n",
        "![example of input and output shapes for a housing price prediction problem](https://raw.githubusercontent.com/mrdbourke/tensorflow-deep-learning/main/images/01-input-and-output-shapes-housing-prices.png)\n",
        "*If you were working on building a machine learning algorithm for predicting housing prices, your inputs may be number of bedrooms, number of bathrooms and number of garages, giving you an input shape of 3 (3 different features). And since you're trying to predict the price of the house, your output shape would be 1.*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PhAIqjrn6olF"
      },
      "source": [
        "## Steps in modelling with TensorFlow\n",
        "\n",
        "Now we know what data we have as well as the input and output shapes, let's see how we'd build a neural network to model it.\n",
        "\n",
        "In TensorFlow, there are typically 3 fundamental steps to creating and training a model.\n",
        "\n",
        "1. **Creating a model** - piece together the layers of a neural network yourself (using the [Functional](https://www.tensorflow.org/guide/keras/functional) or [Sequential API](https://www.tensorflow.org/api_docs/python/tf/keras/Sequential)) or import a previously built model (known as transfer learning).\n",
        "2. **Compiling a model** - defining how a models performance should be measured (loss/metrics) as well as defining how it should improve (optimizer). \n",
        "3. **Fitting a model** - letting the model try to find patterns in the data (how does `X` get to `y`). \n",
        "\n",
        "Let's see these in action using the [Keras Sequential API](https://www.tensorflow.org/api_docs/python/tf/keras/Sequential) to build a model for our regression data. And then we'll step through each.\n",
        "\n",
        "> **Note:** If you're using [TensorFlow 2.7.0](https://github.com/tensorflow/tensorflow/releases/tag/v2.7.0)+, the `fit()` function no longer upscales input data to go from `(batch_size, )` to `(batch_size, 1)`. To fix this, you'll need to expand the dimension of input data using `tf.expand_dims(input_data, axis=-1)`.\n",
        ">\n",
        "> In our case, this means instead of using `model.fit(X, y, epochs=5)`, use `model.fit(tf.expand_dims(X, axis=-1), y, epochs=5)`. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P9jj-OE16yCn"
      },
      "source": [
        "# Set random seed\n",
        "tf.random.set_seed(42)\n",
        "\n",
        "# Create a model using the Sequential API\n",
        "model = tf.keras.Sequential([\n",
        "  tf.keras.layers.Dense(1)\n",
        "])\n",
        "\n",
        "# Compile the model\n",
        "model.compile(loss=tf.keras.losses.mae, # mae is short for mean absolute error\n",
        "              optimizer=tf.keras.optimizers.SGD(), # SGD is short for stochastic gradient descent\n",
        "              metrics=[\"mae\"])\n",
        "\n",
        "# Fit the model\n",
        "# model.fit(X, y, epochs=5) # this will break with TensorFlow 2.7.0+\n",
        "model.fit(tf.expand_dims(X, axis=-1), y, epochs=5)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bbjCwkEtetB9"
      },
      "source": [
        "Boom!\n",
        "\n",
        "We've just trained a model to figure out the patterns between `X` and `y`.\n",
        "\n",
        "How do you think it went?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VWLpG2U3erWo"
      },
      "source": [
        "# Check out X and y\n",
        "X, y"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0ZpkaI_Oe6no"
      },
      "source": [
        "What do you think the outcome should be if we passed our model an `X` value of 17.0?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X86cD66Qeo-8"
      },
      "source": [
        "# Make a prediction with the model\n",
        "model.predict([17.0])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2YoKIqhffq33"
      },
      "source": [
        "It doesn't go very well... it should've output something close to 27.0.\n",
        "\n",
        "> 🤔 **Question:** What's Keras? I thought we were working with TensorFlow but every time we write TensorFlow code, `keras` comes after `tf` (e.g. `tf.keras.layers.Dense()`)?\n",
        "\n",
        "Before TensorFlow 2.0+, [Keras](https://keras.io/) was an API designed to be able to build deep learning models with ease. Since TensorFlow 2.0+, its functionality has been tightly integrated within the TensorFlow library."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qAPk1T3xgOm4"
      },
      "source": [
        "## Improving a model\n",
        "\n",
        "How do you think you'd improve upon our current model?\n",
        "\n",
        "If you guessed by tweaking some of the things we did above, you'd be correct.\n",
        "\n",
        "To improve our model, we alter almost every part of the 3 steps we went through before.\n",
        "\n",
        "1. **Creating a model** - here you might want to add more layers, increase the number of hidden units (also called neurons) within each layer, change the activation functions of each layer.\n",
        "2. **Compiling a model** - you might want to choose optimization function or perhaps change the **learning rate** of the optimization function.\n",
        "3. **Fitting a model** - perhaps you could fit a model for more **epochs** (leave it training for longer) or on more data (give the model more examples to learn from).\n",
        "\n",
        "![various options you can use to improve a neural network model](https://raw.githubusercontent.com/mrdbourke/tensorflow-deep-learning/main/images/02-improving-a-model-from-model-perspective.png)\n",
        "*There are many different ways to potentially improve a neural network. Some of the most common include: increasing the number of layers (making the network deeper), increasing the number of hidden units (making the network wider) and changing the learning rate. Because these values are all human-changeable, they're referred to as [hyperparameters](https://en.wikipedia.org/wiki/Hyperparameter_(machine_learning)) and the practice of trying to find the best hyperparameters is referred to as [hyperparameter tuning](https://en.wikipedia.org/wiki/Hyperparameter_optimization).*\n",
        "\n",
        "Woah. We just introduced a bunch of possible steps. The important thing to remember is how you alter each of these will depend on the problem you're working on.\n",
        "\n",
        "And the good thing is, over the next few problems, we'll get hands-on with all of them.\n",
        "\n",
        "For now, let's keep it simple, all we'll do is train our model for longer (everything else will stay the same)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MI0LammMgWcN"
      },
      "source": [
        "# Set random seed\n",
        "tf.random.set_seed(42)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### ❓ Q3. Create a model just like above (with one layer that has one neuron) and compile it"
      ],
      "metadata": {
        "id": "3KC1sYHWRqQf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%run_and_save student_input.py\n",
        "model = ____ # YOUR CODE HERE\n",
        "\n",
        "model.compile(loss=____, # YOUR CODE HERE\n",
        "              optimizer=____, # YOUR CODE HERE\n",
        "              metrics=['mae'])\n",
        "\n",
        "layer_name = model.layers[0].name[0:5]\n",
        "units = model.layers[0].units\n",
        "loss = model.loss\n",
        "optimizer = model.optimizer.get_config()['name']"
      ],
      "metadata": {
        "id": "FqnJxJTGRxLU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "q_3.submit_answer()\n",
        "\n",
        "Ex().check_object('layer_name').has_equal_value()\n",
        "Ex().check_object('units').has_equal_value()\n",
        "Ex().check_object('loss').has_equal_value()\n",
        "Ex().check_object('optimizer').has_equal_value()\n",
        "print('Success!')"
      ],
      "metadata": {
        "id": "URbq4VbGR3xQ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Lines below will give you a hint or solution code (uncomment the line and run it)\n",
        "# q_3.hint()\n",
        "# q_3.solution()"
      ],
      "metadata": {
        "id": "aGmTWwGLSBE4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now that we have compiled the model, let's fit it for longer to see if that improves the results!"
      ],
      "metadata": {
        "id": "Q2kM0NM_da0P"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Fit model (this time we'll train for longer)\n",
        "model.fit(tf.expand_dims(X, axis=-1), y, epochs=100) # train for 100 epochs not 10"
      ],
      "metadata": {
        "id": "67iF_GLsRohf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1CIKSm7filgj"
      },
      "source": [
        "You might've noticed the loss value decrease from before (and keep decreasing as the number of epochs gets higher). \n",
        "\n",
        "What do you think this means for when we make a prediction with our model?\n",
        "\n",
        "How about we try predict on 17.0 again?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_YcacZsfi4zZ"
      },
      "source": [
        "# Remind ourselves of what X and y are\n",
        "X, y"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### ❓ Q4. Predict what y would be if X was 17.0"
      ],
      "metadata": {
        "id": "--ao9nTudzN8"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e6k5V08ZivNO"
      },
      "source": [
        "prediction = ____ # YOUR CODE HERE - you should get an answer somewhat close to 27\n",
        "print(prediction)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# q_4.solution()"
      ],
      "metadata": {
        "id": "AlThKgbJeJgy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sYi5V8ZYi_dy"
      },
      "source": [
        "Much better! \n",
        "\n",
        "We got closer this time. But we could still be better.\n",
        "\n",
        "Now we've trained a model, how could we evaluate it?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HvC98q_h6zvG"
      },
      "source": [
        "## Evaluating a model \n",
        "\n",
        "A typical workflow you'll go through when building neural networks is:\n",
        "\n",
        "```\n",
        "Build a model -> evaluate it -> build (tweak) a model -> evaulate it -> build (tweak) a model -> evaluate it...\n",
        "```\n",
        "\n",
        "The tweaking comes from maybe not building a model from scratch but adjusting an existing one.\n",
        "\n",
        "### Visualize, visualize, visualize\n",
        "\n",
        "When it comes to evaluation, you'll want to remember the words: \"visualize, visualize, visualize.\" \n",
        "\n",
        "This is because you're probably better looking at something (doing) than you are thinking about something.\n",
        "\n",
        "It's a good idea to visualize:\n",
        "* **The data** - what data are you working with? What does it look like?\n",
        "* **The model itself** - what does the architecture look like? What are the different shapes?\n",
        "* **The training of a model** - how does a model perform while it learns?\n",
        "* **The predictions of a model** - how do the predictions of a model line up against the ground truth (the original labels)?\n",
        "\n",
        "Let's start by visualizing the model.\n",
        "\n",
        "But first, we'll create a little bit of a bigger dataset and a new model we can use (it'll be the same as before, but the more practice the better).\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "srxuqbeYopns"
      },
      "source": [
        "# Make a bigger dataset\n",
        "X = np.arange(-100, 100, 4)\n",
        "X"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pQcC0nSko3kJ"
      },
      "source": [
        "# Make labels for the dataset (adhering to the same pattern as before)\n",
        "y = np.arange(-90, 110, 4)\n",
        "y"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KNIw5tgGpKjb"
      },
      "source": [
        "Since $y=X+10$, we could make the labels like so:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ACgbmrAOpJwW"
      },
      "source": [
        "# Same result as above\n",
        "y = X + 10\n",
        "y"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ax3MnQDupeBp"
      },
      "source": [
        "## Split data into training/test set\n",
        "\n",
        "One of the other most common and important steps in a machine learning project is creating a training and test set (and when required, a validation set).\n",
        "\n",
        "Each set serves a specific purpose:\n",
        "* **Training set** - the model learns from this data, which is typically 70-80% of the total data available (like the course materials you study during the semester).\n",
        "* **Validation set** - the model gets tuned on this data, which is typically 10-15% of the total data available (like the practice exam you take before the final exam).\n",
        "* **Test set** - the model gets evaluated on this data to test what it has learned, it's typically 10-15% of the total data available (like the final exam you take at the end of the semester).\n",
        "\n",
        "For now, we'll just use a training and test set, this means we'll have a dataset for our model to learn on as well as be evaluated on.\n",
        "\n",
        "We can create them by splitting our `X` and `y` arrays.\n",
        "\n",
        "> 🔑 **Note:** When dealing with real-world data, this step is typically done right at the start of a project (the test set should always be kept separate from all other data). We want our model to learn on training data and then evaluate it on test data to get an indication of how well it **generalizes** to unseen examples."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5G0RDMnZrgvK"
      },
      "source": [
        "# Check how many samples we have\n",
        "len(X)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4Q9ptcQkrGfO"
      },
      "source": [
        "# Split data into train and test sets\n",
        "X_train = X[:40] # first 40 examples (80% of data)\n",
        "y_train = y[:40]\n",
        "\n",
        "X_test = X[40:] # last 10 examples (20% of data)\n",
        "y_test = y[40:]\n",
        "\n",
        "len(X_train), len(X_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Rz2cIdECsLH5"
      },
      "source": [
        "## Visualizing the data\n",
        "\n",
        "Now we've got our training and test data, it's a good idea to visualize it.\n",
        "\n",
        "Let's plot it with some nice colours to differentiate what's what."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "os30CXBHsOAH"
      },
      "source": [
        "plt.figure(figsize=(10, 7))\n",
        "# Plot training data in blue\n",
        "plt.scatter(X_train, y_train, c='b', label='Training data')\n",
        "# Plot test data in green\n",
        "plt.scatter(X_test, y_test, c='g', label='Testing data')\n",
        "# Show the legend\n",
        "plt.legend();"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cxRcHZFgtS_B"
      },
      "source": [
        "Beautiful! Any time you can visualize your data, your model, your anything, it's a good idea. \n",
        "\n",
        "With this graph in mind, what we'll be trying to do is build a model which learns the pattern in the blue dots (`X_train`) to draw the green dots (`X_test`).\n",
        "\n",
        "Time to build a model. We'll make the exact same one from before (the one we trained for longer)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4qpe0eSStSm-"
      },
      "source": [
        "# Set random seed\n",
        "tf.random.set_seed(42)\n",
        "\n",
        "# Create a model (same as above)\n",
        "model = tf.keras.Sequential([\n",
        "  tf.keras.layers.Dense(1)\n",
        "])\n",
        "\n",
        "# Compile model (same as above)\n",
        "model.compile(loss=tf.keras.losses.mae,\n",
        "              optimizer=tf.keras.optimizers.SGD(),\n",
        "              metrics=[\"mae\"])\n",
        "\n",
        "# Fit model (same as above)\n",
        "#model.fit(X_train, y_train, epochs=100) # commented out on purpose (not fitting it just yet)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hc2RHCCfqlAc"
      },
      "source": [
        "## Visualizing the model\n",
        "\n",
        "After you've built a model, you might want to take a look at it (especially if you haven't built many before).\n",
        "\n",
        "You can take a look at the layers and shapes of your model by calling [`summary()`](https://www.tensorflow.org/api_docs/python/tf/keras/Model#summary) on it.\n",
        "\n",
        "> 🔑 **Note:** Visualizing a model is particularly helpful when you run into input and output shape mismatches."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OlgJj0cFwChH"
      },
      "source": [
        "# Doesn't work (model not fit/built)\n",
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SJj6KLe8xsyF"
      },
      "source": [
        "Ahh, the cell above errors because we haven't fit our built our model.\n",
        "\n",
        "We also haven't told it what input shape it should be expecting.\n",
        "\n",
        "Remember above, how we discussed the input shape was just one number?\n",
        "\n",
        "We can let our model know the input shape of our data using the `input_shape` parameter to the first layer (usually if `input_shape` isn't defined, Keras tries to figure it out automatically)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zMXKFtFBuWgJ"
      },
      "source": [
        "# Set random seed\n",
        "tf.random.set_seed(42)\n",
        "\n",
        "# Create a model (same as above)\n",
        "model = tf.keras.Sequential([\n",
        "  tf.keras.layers.Dense(1, input_shape=[1]) # define the input_shape to our model\n",
        "])\n",
        "\n",
        "# Compile model (same as above)\n",
        "model.compile(loss=tf.keras.losses.mae,\n",
        "              optimizer=tf.keras.optimizers.SGD(),\n",
        "              metrics=[\"mae\"])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "24Q325x2yCoi"
      },
      "source": [
        "# This will work after specifying the input shape\n",
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jdurWKlryl6b"
      },
      "source": [
        "Calling `summary()` on our model shows us the layers it contains, the output shape and the number of parameters.\n",
        "* **Total params** - total number of parameters in the model.\n",
        "* **Trainable parameters** - these are the parameters (patterns) the model can update as it trains.\n",
        "* **Non-trainable parameters** - these parameters aren't updated during training (this is typical when you bring in the already learned patterns from other models during transfer learning).\n",
        "\n",
        "> 📖 **Resource:** For a more in-depth overview of the trainable parameters within a layer, check out [MIT's introduction to deep learning video](https://youtu.be/njKP3FqW3Sk).\n",
        "\n",
        "> 🛠 **Exercise:** Try playing around with the number of hidden units in the `Dense` layer (e.g. `Dense(2)`, `Dense(3)`). How does this change the Total/Trainable params? Investigate what's causing the change.\n",
        "\n",
        "For now, all you need to think about these parameters is that their learnable patterns in the data.\n",
        "\n",
        "Let's fit our model to the training data."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### ❓ Q5. Fit the compiled model"
      ],
      "metadata": {
        "id": "SWBOGEkk1feB"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-Kywg4q9u051"
      },
      "source": [
        "# Fit the model to the training data, 100 epochs, and add verbose=0 to the arguments\n",
        "____ # YOUR CODE HERE"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# q_5.solution()"
      ],
      "metadata": {
        "id": "xqieRevN15tm"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### ❓ Q6. Check model summary"
      ],
      "metadata": {
        "id": "BIuFOiu42Ymq"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-jWuOwj961ri"
      },
      "source": [
        "____ # YOUR CODE HERE"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# q_6.solution()"
      ],
      "metadata": {
        "id": "Y5p3smde2g2m"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RrrkHEkMYh5A"
      },
      "source": [
        "Alongside summary, you can also view a 2D plot of the model using [`plot_model()`](https://www.tensorflow.org/api_docs/python/tf/keras/utils/plot_model)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "liIg5WqDfdK4"
      },
      "source": [
        "from tensorflow.keras.utils import plot_model\n",
        "\n",
        "plot_model(model, show_shapes=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ygw7V8OVYxQS"
      },
      "source": [
        "In our case, the model we used only has an input and an output but visualizing more complicated models can be very helpful for debugging."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gebj2eptqwg8"
      },
      "source": [
        "## Visualizing the predictions\n",
        "\n",
        "Now we've got a trained model, let's visualize some predictions.\n",
        "\n",
        "To visualize predictions, it's always a good idea to plot them against the ground truth labels.\n",
        "\n",
        "Often you'll see this in the form of `y_test` vs. `y_pred` (ground truth vs. predictions).\n",
        "\n",
        "First, we'll make some predictions on the test data (`X_test`), remember the model has never seen the test data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RRzj7LJMYftb"
      },
      "source": [
        "# Make predictions\n",
        "y_preds = model.predict(X_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_hKpW-KOZiAW"
      },
      "source": [
        "# View the predictions\n",
        "y_preds"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aPRaFncaZnT-"
      },
      "source": [
        "Okay, we get a list of numbers but how do these compare to the ground truth labels?\n",
        "\n",
        "Let's build a plotting function to find out.\n",
        "\n",
        "> 🔑 **Note:** If you think you're going to be visualizing something a lot, it's a good idea to functionize it so you can use it later."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "56euC69rZvNJ"
      },
      "source": [
        "def plot_predictions(train_data=X_train, \n",
        "                     train_labels=y_train, \n",
        "                     test_data=X_test, \n",
        "                     test_labels=y_test, \n",
        "                     predictions=y_preds):\n",
        "  \"\"\"\n",
        "  Plots training data, test data and compares predictions.\n",
        "  \"\"\"\n",
        "  plt.figure(figsize=(10, 7))\n",
        "  # Plot training data in blue\n",
        "  plt.scatter(train_data, train_labels, c=\"b\", label=\"Training data\")\n",
        "  # Plot test data in green\n",
        "  plt.scatter(test_data, test_labels, c=\"g\", label=\"Testing data\")\n",
        "  # Plot the predictions in red (predictions were made on the test data)\n",
        "  plt.scatter(test_data, predictions, c=\"r\", label=\"Predictions\")\n",
        "  # Show the legend\n",
        "  plt.legend();"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fug5_B6Ab7Ah"
      },
      "source": [
        "plot_predictions(train_data=X_train,\n",
        "                 train_labels=y_train,\n",
        "                 test_data=X_test,\n",
        "                 test_labels=y_test,\n",
        "                 predictions=y_preds)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H63NPcgPcwZV"
      },
      "source": [
        "From the plot we can see our predictions aren't totally outlandish but they definitely aren't anything special either."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "orAfIWOrgRDA"
      },
      "source": [
        "## Evaluating predictions\n",
        "\n",
        "Alongside visualizations, evaulation metrics are your alternative best option for evaluating your model.\n",
        "\n",
        "Depending on the problem you're working on, different models have different evaluation metrics. \n",
        "\n",
        "Two of the main metrics used for regression problems are:\n",
        "* **Mean absolute error (MAE)** - the mean difference between each of the predictions.\n",
        "* **Mean squared error (MSE)** - the squared mean difference between of the predictions (use if larger errors are more detrimental than smaller errors).\n",
        "\n",
        "The lower each of these values, the better.\n",
        "\n",
        "You can also use [`model.evaluate()`](https://www.tensorflow.org/api_docs/python/tf/keras/Model#evaluate) which will return the loss of the model as well as any metrics setup during the compile step."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DPgTdF3ddxiY"
      },
      "source": [
        "# Evaluate the model on the test set\n",
        "model.evaluate(X_test, y_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DAXIRyVzegFd"
      },
      "source": [
        "In our case, since we used MAE for the loss function as well as MAE for the metrics, `model.evaulate()` returns them both.\n",
        "\n",
        "TensorFlow also has built in functions for MSE and MAE.\n",
        "\n",
        "For many evaluation functions, the premise is the same: compare predictions to the ground truth labels."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gqoMQ0dJeD2S"
      },
      "source": [
        "# Calculate the mean absolute error\n",
        "mae = tf.metrics.mean_absolute_error(y_true=y_test, \n",
        "                                     y_pred=y_preds)\n",
        "mae"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7pYp3XNYfkyC"
      },
      "source": [
        "Huh? That's strange, MAE should be a single output.\n",
        "\n",
        "Instead, we get 10 values.\n",
        "\n",
        "This is because our `y_test` and `y_preds` tensors are different shapes."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zeUHwOc2gIV4"
      },
      "source": [
        "# Check the test label tensor values\n",
        "y_test"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3aJwlTxugPyc"
      },
      "source": [
        "# Check the predictions tensor values (notice the extra square brackets)\n",
        "y_preds"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xolZ-lmge_ES"
      },
      "source": [
        "# Check the tensor shapes\n",
        "y_test.shape, y_preds.shape"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eW1qleu5gHyZ"
      },
      "source": [
        "Remember how we discussed dealing with different input and output shapes is one the most common issues you'll come across, this is one of those times.\n",
        "\n",
        "But not to worry.\n",
        "\n",
        "We can fix it using [`squeeze()`](https://www.tensorflow.org/api_docs/python/tf/squeeze), it'll remove the the `1` dimension from our `y_preds` tensor, making it the same shape as `y_test`.\n",
        "\n",
        "> 🔑 **Note:** If you're comparing two tensors, it's important to make sure they're the right shape(s) (you won't always have to manipulate the shapes, but always be on the look out, *many* errors are the result of mismatched tensors, especially mismatched input and output shapes)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tVtMyw70g4aF"
      },
      "source": [
        "# Shape before squeeze()\n",
        "y_preds.shape"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qnYaBnCng-Nq"
      },
      "source": [
        "# Shape after squeeze()\n",
        "y_preds.squeeze().shape"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HxvVeD64hEX8"
      },
      "source": [
        "# What do they look like?\n",
        "y_test, y_preds.squeeze()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mfUCIeHyhLk7"
      },
      "source": [
        "Okay, now we know how to make our `y_test` and `y_preds` tenors the same shape, let's use our evaluation metrics."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TvjY6GIJvXBO"
      },
      "source": [
        "# Calcuate the MAE\n",
        "mae = tf.metrics.mean_absolute_error(y_true=y_test, \n",
        "                                     y_pred=y_preds.squeeze()) # use squeeze() to make same shape\n",
        "mae"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### ❓ Q7. Calculate the MSE"
      ],
      "metadata": {
        "id": "sh03N3Fb3mf4"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EwA9nGJzvbMT"
      },
      "source": [
        "mse = ____ # YOUR CODE HERE\n",
        "mse"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# q_7.hint()\n",
        "# q_7.solution()"
      ],
      "metadata": {
        "id": "Yi2wa0CV3ytM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X06oo_CIjPbL"
      },
      "source": [
        "We can also calculate the MAE using pure TensorFlow functions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YxiD6-QBYSzd"
      },
      "source": [
        "# Returns the same as tf.metrics.mean_absolute_error()\n",
        "tf.reduce_mean(tf.abs(y_test-y_preds.squeeze()))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rmEho4lYofOa"
      },
      "source": [
        "Again, it's a good idea to functionize anything you think you might use over again (or find yourself using over and over again).\n",
        "\n",
        "Let's make functions for our evaluation metrics."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bs1Z2jgNol5f"
      },
      "source": [
        "def mae(y_test, y_pred):\n",
        "  \"\"\"\n",
        "  Calculuates mean absolute error between y_test and y_preds.\n",
        "  \"\"\"\n",
        "  return tf.metrics.mean_absolute_error(y_test,\n",
        "                                        y_pred)\n",
        "  \n",
        "def mse(y_test, y_pred):\n",
        "  \"\"\"\n",
        "  Calculates mean squared error between y_test and y_preds.\n",
        "  \"\"\"\n",
        "  return tf.metrics.mean_squared_error(y_test,\n",
        "                                       y_pred)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zub5zK7bcl40"
      },
      "source": [
        "## Running experiments to improve a model\n",
        "\n",
        "After seeing the evaluation metrics and the predictions your model makes, it's likely you'll want to improve it.\n",
        "\n",
        "Again, there are many different ways you can do this, but 3 of the main ones are:\n",
        "1. **Get more data** - get more examples for your model to train on (more opportunities to learn patterns).\n",
        "2. **Make your model larger (use a more complex model)** - this might come in the form of more layers or more hidden units in each layer.\n",
        "3. **Train for longer** - give your model more of a chance to find the patterns in the data.\n",
        "\n",
        "Since we created our dataset, we could easily make more data but this isn't always the case when you're working with real-world datasets.\n",
        "\n",
        "So let's take a look at how we can improve our model using 2 and 3.\n",
        "\n",
        "To do so, we'll build 3 models and compare their results:\n",
        "1. `model_1` - same as original model, 1 layer, trained for 100 epochs.\n",
        "2. `model_2` - 2 layers, trained for 100 epochs.\n",
        "3. `model_3` - 2 layers, trained for 500 epochs.\n",
        "\n",
        "**Build `model_1`**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "StVHIIM9csyS"
      },
      "source": [
        "# Set random seed\n",
        "tf.random.set_seed(42)\n",
        "\n",
        "# Replicate original model\n",
        "model_1 = tf.keras.Sequential([\n",
        "  tf.keras.layers.Dense(1)\n",
        "])\n",
        "\n",
        "# Compile the model\n",
        "model_1.compile(loss=tf.keras.losses.mae,\n",
        "                optimizer=tf.keras.optimizers.SGD(),\n",
        "                metrics=['mae'])\n",
        "\n",
        "# Fit the model\n",
        "model_1.fit(tf.expand_dims(X_train, axis=-1), y_train, epochs=100)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A-Da56xspOrY"
      },
      "source": [
        "# Make and plot predictions for model_1\n",
        "y_preds_1 = model_1.predict(X_test)\n",
        "plot_predictions(predictions=y_preds_1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EXxHBAtHoSh2"
      },
      "source": [
        "# Calculate model_1 metrics\n",
        "mae_1 = mae(y_test, y_preds_1.squeeze()).numpy()\n",
        "mse_1 = mse(y_test, y_preds_1.squeeze()).numpy()\n",
        "mae_1, mse_1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XXELOpdBrE9_"
      },
      "source": [
        "**Build `model_2`**\n",
        "\n",
        "This time we'll add an extra dense layer (so now our model will have 2 layers) whilst keeping everything else the same.\n",
        "\n",
        "### ❓ Q8. Build a second model but with 2 dense layers"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "05vcgEP3rEFi"
      },
      "source": [
        "# Set random seed\n",
        "tf.random.set_seed(42)\n",
        "\n",
        "# Replicate model_1 but have 2 dense layers\n",
        "model_2 = tf.keras.Sequential([\n",
        "  ____ # YOUR CODE HERE\n",
        "])\n",
        "\n",
        "# Compile the model\n",
        "model_2.compile(loss=tf.keras.losses.mae,\n",
        "                optimizer=tf.keras.optimizers.SGD(),\n",
        "                metrics=['mae'])\n",
        "\n",
        "# Fit the model\n",
        "model_2.fit(tf.expand_dims(X_train, axis=-1), y_train, epochs=100, verbose=0) # set verbose to 0 for less output"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# q_8.solution()"
      ],
      "metadata": {
        "id": "Y4FGBdWe4_YF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### ❓ Q9. Make and plot predictions for model_2"
      ],
      "metadata": {
        "id": "GUxSiPTt5aew"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9xCbDcoDraux"
      },
      "source": [
        "# Make and plot predictions for model_2\n",
        "y_preds_2 = ____ # YOUR CODE HERE\n",
        "plot_predictions(predictions=y_preds_2)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# q_9.solution()"
      ],
      "metadata": {
        "id": "Pb1APA1-5vo1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GxMYu5Frri6o"
      },
      "source": [
        "Woah, that's looking better already! And all it took was an extra layer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9LnPuGaBrcrP"
      },
      "source": [
        "# Calculate model_2 metrics\n",
        "mae_2 = mae(y_test, y_preds_2.squeeze()).numpy()\n",
        "mse_2 = mse(y_test, y_preds_2.squeeze()).numpy()\n",
        "mae_2, mse_2"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R8i9yfQGrwHx"
      },
      "source": [
        "**Build `model_3`**\n",
        "\n",
        "For our 3rd model, we'll keep everything the same as `model_2` except this time we'll train for longer (500 epochs instead of 100).\n",
        "\n",
        "This will give our model more of a chance to learn the patterns in the data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ABGwQFsbrvUS"
      },
      "source": [
        "# Set random seed\n",
        "tf.random.set_seed(42)\n",
        "\n",
        "# Replicate model_2\n",
        "model_3 = tf.keras.Sequential([\n",
        "  tf.keras.layers.Dense(1),\n",
        "  tf.keras.layers.Dense(1)\n",
        "])\n",
        "\n",
        "# Compile the model\n",
        "model_3.compile(loss=tf.keras.losses.mae,\n",
        "                optimizer=tf.keras.optimizers.SGD(),\n",
        "                metrics=['mae'])\n",
        "\n",
        "# Fit the model (this time for 500 epochs, not 100)\n",
        "model_3.fit(tf.expand_dims(X_train, axis=-1), y_train, epochs=500, verbose=0) # set verbose to 0 for less output"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jEz4bVmasbFk"
      },
      "source": [
        "# Make and plot predictions for model_3\n",
        "y_preds_3 = model_3.predict(X_test)\n",
        "plot_predictions(predictions=y_preds_3)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "38Ki8ZHcsztG"
      },
      "source": [
        "Strange, we trained for longer but our model performed worse?\n",
        "\n",
        "As it turns out, our model might've trained too long and has thus resulted in worse results (we'll see ways to prevent training for too long later on)."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### ❓ Q10. Calculate model_3 metrics\n",
        "\n",
        "Note: make sure mae_3 and mse_3 are numpy numbers not tf tensors"
      ],
      "metadata": {
        "id": "Me_Kw-O56Uzr"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BPTUcFe4sbfk"
      },
      "source": [
        "%%run_and_save student_input.py\n",
        "mae_3 = mae(y_test, y_preds_3.squeeze()).numpy()\n",
        "mse_3 = mse(y_test, y_preds_3.squeeze()).numpy()\n",
        "print(mae_3, mse_3)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "q_10.submit_answer()\n",
        "\n",
        "Ex().check_object('mae_3').has_equal_value()\n",
        "Ex().check_object('mse_3').has_equal_value()\n",
        "\n",
        "print('Success!')"
      ],
      "metadata": {
        "id": "y045DjTf8L52"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# q_10.hint()\n",
        "# q_10.solution()"
      ],
      "metadata": {
        "id": "W9LYLcjc_c1W"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UPEeM3UsrxGB"
      },
      "source": [
        "## Comparing results\n",
        "\n",
        "Now we've got results for 3 similar but slightly different results, let's compare them."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mw5RZk-BqLZd"
      },
      "source": [
        "model_results = [[\"model_1\", mae_1, mse_1],\n",
        "                 [\"model_2\", mae_2, mse_2],\n",
        "                 [\"model_3\", mae_3, mae_3]]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ip7bKH83p5X0"
      },
      "source": [
        "import pandas as pd\n",
        "all_results = pd.DataFrame(model_results, columns=[\"model\", \"mae\", \"mse\"])\n",
        "all_results"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o_AtUiwuuLGo"
      },
      "source": [
        "From our experiments, it looks like `model_2` performed the best.\n",
        "\n",
        "And now, you might be thinking, \"wow, comparing models is tedious...\" and it definitely can be, we've only compared 3 models here. \n",
        "\n",
        "But this is part of what machine learning modelling is about, trying many different combinations of models and seeing which performs best.\n",
        "\n",
        "Each model you build is a small experiment. \n",
        "\n",
        "> 🔑 **Note:** One of your main goals should be to minimize the time between your experiments. The more experiments you do, the more things you'll figure out which don't work and in turn, get closer to figuring out what does work. Remember the machine learning practitioner's motto: \"experiment, experiment, experiment\".\n",
        "\n",
        "Another thing you'll also find is what you thought may work (such as training a model for longer) may not always work and the exact opposite is also often the case.\n",
        "\n",
        "## Tracking your experiments\n",
        "\n",
        "One really good habit to get into is tracking your modelling experiments to see which perform better than others.\n",
        "\n",
        "We've done a simple version of this above (keeping the results in different variables).\n",
        "\n",
        "> 📖 **Resource:** But as you build more models, you'll want to look into using tools such as:\n",
        "* [**TensorBoard**](https://tensorboard.dev/) - a component of the TensorFlow library to help track modelling experiments (we'll see this later).\n",
        "* [**Weights & Biases**](https://www.wandb.com/) - a tool for tracking all kinds of machine learning experiments (the good news for Weights & Biases is it plugs into TensorBoard)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fe5DgNbX6192"
      },
      "source": [
        "## Saving a model\n",
        "\n",
        "Once you've trained a model and found one which performs to your liking, you'll probably want to save it for use elsewhere (like a web application or mobile device).\n",
        "\n",
        "You can save a TensorFlow/Keras model using [`model.save()`](https://www.tensorflow.org/tutorials/keras/save_and_load#save_the_entire_model).\n",
        "\n",
        "There are two ways to save a model in TensorFlow:\n",
        "1. The [SavedModel format](https://www.tensorflow.org/tutorials/keras/save_and_load#savedmodel_format) (default).\n",
        "2. The [HDF5 format](https://www.tensorflow.org/tutorials/keras/save_and_load#hdf5_format).\n",
        "\n",
        "The main difference between the two is the SavedModel is automatically able to save custom objects (such as special layers) without additional modifications when loading the model back in.\n",
        "\n",
        "Which one should you use?\n",
        "\n",
        "It depends on your situation but the SavedModel format will suffice most of the time.\n",
        "\n",
        "Both methods use the same method call."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gg0jD2cUoPsg"
      },
      "source": [
        "# Save a model using the SavedModel format\n",
        "model_2.save('best_model_SavedModel_format')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dsCpDYrU7D1j"
      },
      "source": [
        "# Check it out - outputs a protobuf binary file (.pb) as well as other files\n",
        "!ls best_model_SavedModel_format"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NGKmWco_SOEU"
      },
      "source": [
        "Now let's save the model in the HDF5 format, we'll use the same method but with a different filename."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "97J6GJMBSM2j"
      },
      "source": [
        "# Save a model using the HDF5 format\n",
        "model_2.save(\"best_model_HDF5_format.h5\") # note the addition of '.h5' on the end"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vB7TmsSGSjdv"
      },
      "source": [
        "# Check it out\n",
        "!ls best_model_HDF5_format.h5"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OGA02tY97EUI"
      },
      "source": [
        "## Loading a model\n",
        "\n",
        "We can load a saved model using the [`load_model()`](https://www.tensorflow.org/api_docs/python/tf/keras/models/load_model) method.\n",
        "\n",
        "Loading a model for the different formats (SavedModel and HDF5) is the same (as long as the pathnames to the particuluar formats are correct)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FzyLIWfs7Fvh"
      },
      "source": [
        "# Load a model from the SavedModel format\n",
        "loaded_saved_model = tf.keras.models.load_model(\"best_model_SavedModel_format\")\n",
        "loaded_saved_model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MfGO0dCQTeQh"
      },
      "source": [
        "Now let's test it out."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7rehN8ZxTy43"
      },
      "source": [
        "# Compare model_2 with the SavedModel version (should return True)\n",
        "model_2_preds = model_2.predict(X_test)\n",
        "saved_model_preds = loaded_saved_model.predict(X_test)\n",
        "mae(y_test, saved_model_preds.squeeze()).numpy() == mae(y_test, model_2_preds.squeeze()).numpy()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "prjkfX6rUZ6a"
      },
      "source": [
        "Loading in from the HDF5 is much the same."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dQfx-bWKUfRQ"
      },
      "source": [
        "# Load a model from the HDF5 format\n",
        "loaded_h5_model = tf.keras.models.load_model(\"best_model_HDF5_format.h5\")\n",
        "loaded_h5_model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L0kT91h-Uru-"
      },
      "source": [
        "# Compare model_2 with the loaded HDF5 version (should return True)\n",
        "h5_model_preds = loaded_h5_model.predict(X_test)\n",
        "mae(y_test, h5_model_preds.squeeze()).numpy() == mae(y_test, model_2_preds.squeeze()).numpy()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### ❓ Bonus Question\n",
        "\n",
        "Try saving and loading model_3"
      ],
      "metadata": {
        "id": "P6uPYnFCDSFV"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ABtsYBDtr5Zz"
      },
      "source": [
        "## Downloading a model (from Google Colab)\n",
        "\n",
        "Say you wanted to get your model from Google Colab to your local machine, you can do one of the following things:\n",
        "* Right click on the file in the files pane and click 'download'.\n",
        "* Use the code below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JV0onjIIr9XC"
      },
      "source": [
        "# Download the model (or any file) from Google Colab\n",
        "from google.colab import files\n",
        "files.download(\"best_model_HDF5_format.h5\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7xpVdMtKw6X4"
      },
      "source": [
        "## A larger example\n",
        "\n",
        "Alright, we've seen the fundamentals of building neural network regression models in TensorFlow.\n",
        "\n",
        "Let's step it up a notch and build a model for a more feature rich datase.\n",
        "\n",
        "More specifically we're going to try predict the cost of medical insurance for individuals based on a number of different parameters such as, `age`, `sex`, `bmi`, `children`, `smoking_status` and `residential_region`.\n",
        "\n",
        "To do, we'll leverage the pubically available [Medical Cost dataset](https://www.kaggle.com/mirichoi0218/insurance) available from Kaggle and [hosted on GitHub](https://github.com/stedy/Machine-Learning-with-R-datasets/blob/master/insurance.csv).\n",
        "\n",
        "> 🔑 **Note:** When learning machine learning paradigms, you'll often go through a series of foundational techniques and then practice them by working with open-source datasets and examples. Just as we're doing now, learn foundations, put them to work with different problems. Every time you work on something new, it's a good idea to search for something like \"problem X example with Python/TensorFlow\" where you substitute X for your problem."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WWK1LBxapgc2"
      },
      "source": [
        "# Import required libraries\n",
        "import tensorflow as tf\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JrnTr5N9blFo"
      },
      "source": [
        "# Read in the insurance dataset\n",
        "insurance = pd.read_csv(\"https://raw.githubusercontent.com/stedy/Machine-Learning-with-R-datasets/master/insurance.csv\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QtXPN7cfb4Nm"
      },
      "source": [
        "# Check out the insurance dataset\n",
        "insurance.head()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s55oIsYv0KkZ"
      },
      "source": [
        "We're going to have to turn the non-numerical columns into numbers (because a neural network can't handle non-numerical inputs).\n",
        "\n",
        "To do so, we'll use the [`get_dummies()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.get_dummies.html) method in pandas.\n",
        "\n",
        "It converts categorical variables (like the `sex`, `smoker` and `region` columns) into numerical variables using one-hot encoding."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MqM_BmPkdon8"
      },
      "source": [
        "# Turn all categories into numbers\n",
        "insurance_one_hot = pd.get_dummies(insurance)\n",
        "insurance_one_hot.head() # view the converted columns"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zOHoPzgqgZPq"
      },
      "source": [
        "Now we'll split data into features (`X`) and labels (`y`)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W_EGj3FxhkAb"
      },
      "source": [
        "# Create X & y values\n",
        "X = insurance_one_hot.drop(\"charges\", axis=1)\n",
        "y = insurance_one_hot[\"charges\"]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-QQFJmDn5ATV"
      },
      "source": [
        "# View features\n",
        "X.head()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kudhkM-0giS1"
      },
      "source": [
        "And create training and test sets. We could do this manually, but to make it easier, we'll leverage the already available [`train_test_split`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) function available from Scikit-Learn."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WPGZGk0jhxCZ"
      },
      "source": [
        "# Create training and test sets\n",
        "from sklearn.model_selection import train_test_split\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, \n",
        "                                                    y, \n",
        "                                                    test_size=0.2, \n",
        "                                                    random_state=42) # set random state for reproducible splits"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W8wEC0FPglnN"
      },
      "source": [
        "### ❓ Q11. Now we can build and fit a model (we'll make it the same as `model_2`)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OCXTmz6oh_T6"
      },
      "source": [
        "# Set random seed\n",
        "tf.random.set_seed(42)\n",
        "\n",
        "# Create a new model (same as model_2)\n",
        "insurance_model = ____ # YOUR CODE HERE\n",
        "\n",
        "# Compile the insurance_model (same as model_2)\n",
        "____ # YOUR CODE HERE\n",
        "\n",
        "# Fit the insurance_model with 100 epochs\n",
        "____ # YOUR CODE HERE"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# q_11.solution()"
      ],
      "metadata": {
        "id": "XmbYisNcAhGo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B1U7LqpKid0r"
      },
      "source": [
        "# Check the results of the insurance model\n",
        "insurance_model.evaluate(X_test, y_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H9doNDToqDru"
      },
      "source": [
        "Our model didn't perform very well, let's try a bigger model.\n",
        "\n",
        "We'll try 3 things:\n",
        "- Increasing the number of layers (2 -> 3).\n",
        "- Increasing the number of units in each layer (except for the output layer).\n",
        "- Changing the optimizer (from SGD to Adam).\n",
        "\n",
        "Everything else will stay the same."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W59EqfqYimnR"
      },
      "source": [
        "# Set random seed\n",
        "tf.random.set_seed(42)\n",
        "\n",
        "# Add an extra layer and increase number of units\n",
        "insurance_model_2 = tf.keras.Sequential([\n",
        "  tf.keras.layers.Dense(100), # 100 units\n",
        "  tf.keras.layers.Dense(10), # 10 units\n",
        "  tf.keras.layers.Dense(1) # 1 unit (important for output layer)\n",
        "])\n",
        "\n",
        "# Compile the model\n",
        "insurance_model_2.compile(loss=tf.keras.losses.mae,\n",
        "                          optimizer=tf.keras.optimizers.Adam(), # Adam works but SGD doesn't \n",
        "                          metrics=['mae'])\n",
        "\n",
        "# Fit the model and save the history (we can plot this)\n",
        "history = insurance_model_2.fit(X_train, y_train, epochs=100, verbose=0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k9IbYWnOqmoT"
      },
      "source": [
        "# Evaluate our larger model\n",
        "insurance_model_2.evaluate(X_test, y_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o9Rf3NosqriS"
      },
      "source": [
        "Much better! Using a larger model and the Adam optimizer results in almost half the error as the previous model.\n",
        "\n",
        "> 🔑 **Note:** For many problems, the [Adam optimizer](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adam) is a great starting choice. See Andrei Karpathy's \"Adam is safe\" point from [*A Recipe for Training Neural Networks*](http://karpathy.github.io/2019/04/25/recipe/) for more. \n",
        "\n",
        "Let's check out the loss curves of our model, we should see a downward trend."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9SE55ANojcF_"
      },
      "source": [
        "# Plot history (also known as a loss curve)\n",
        "pd.DataFrame(history.history).plot()\n",
        "plt.ylabel(\"loss\")\n",
        "plt.xlabel(\"epochs\");"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ckHqtn0srQ5q"
      },
      "source": [
        "From this, it looks like our model's loss (and MAE) were both still decreasing (in our case, MAE and loss are the same, hence the lines in the plot overlap eachother).\n",
        "\n",
        "What this tells us is the loss might go down if we try training it for longer.\n",
        "\n",
        "> 🤔 **Question:** How long should you train for? \n",
        "\n",
        "> It depends on what problem you're working on. Sometimes training won't take very long, other times it'll take longer than you expect. A common method is to set your model training for a very long time (e.g. 1000's of epochs) but set it up with an [EarlyStopping callback](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping) so it stops automatically when it stops improving. We'll see this in another module.\n",
        "\n",
        "Let's train the same model as above for a little longer. We can do this but calling fit on it again."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ucs62jV8jl6N"
      },
      "source": [
        "# Try training for a little longer (100 more epochs)\n",
        "history_2 = insurance_model_2.fit(X_train, y_train, epochs=100, verbose=0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L2FZA1z1sxxs"
      },
      "source": [
        "How did the extra training go?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cxtiYB3qs0PZ"
      },
      "source": [
        "# Evaluate the model trained for 200 total epochs\n",
        "insurance_model_2_loss, insurance_model_2_mae = insurance_model_2.evaluate(X_test, y_test)\n",
        "insurance_model_2_loss, insurance_model_2_mae"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MzUySYEOs-O_"
      },
      "source": [
        "Boom! Training for an extra 100 epochs we see about a 10% decrease in error.\n",
        "\n",
        "How does the visual look?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BtYKSLeQjuzL"
      },
      "source": [
        "# Plot the model trained for 200 total epochs loss curves\n",
        "pd.DataFrame(history_2.history).plot()\n",
        "plt.ylabel(\"loss\")\n",
        "plt.xlabel(\"epochs\"); # note: epochs will only show 100 since we overrid the history variable"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HS8xYkvIuVZZ"
      },
      "source": [
        "## Preprocessing data (normalization and standardization)\n",
        "\n",
        "A common practice when working with neural networks is to make sure all of the data you pass to them is in the range 0 to 1.\n",
        "\n",
        "This practice is called **normalization** (scaling all values from their original range to, e.g. between 0 and 100,000 to be between 0 and 1).\n",
        "\n",
        "There is another process call **standardization** which converts all of your data to unit variance and 0 mean.\n",
        "\n",
        "These two practices are often part of a preprocessing pipeline (a series of functions to prepare your data for use with neural networks).\n",
        "\n",
        "Knowing this, some of the major steps you'll take to preprocess your data for a neural network include:\n",
        "* Turning all of your data to numbers (a neural network can't handle strings).\n",
        "* Making sure your data is in the right shape (verifying input and output shapes).\n",
        "* [**Feature scaling**](https://scikit-learn.org/stable/modules/preprocessing.html#preprocessing-scaler):\n",
        "    * Normalizing data (making sure all values are between 0 and 1). This is done by subtracting the minimum value then dividing by the maximum value minus the minmum. This is also referred to as min-max scaling.\n",
        "    * Standardization (making sure all values have a mean of 0 and a variance of 1). This is done by substracting the mean value from the target feature and then dividing it by the standard deviation.\n",
        "    * Which one should you use?\n",
        "      * **With neural networks you'll tend to favour normalization** as they tend to prefer values between 0 and 1 (you'll see this espcially with image processing), however, you'll often find a neural network can perform pretty well with minimal feature scaling.\n",
        "\n",
        "> 📖 **Resource:** For more on preprocessing data, I'd recommend reading the following resources:\n",
        "* [Scikit-Learn's documentation on preprocessing data](https://scikit-learn.org/stable/modules/preprocessing.html#preprocessing-data).\n",
        "* [Scale, Standardize or Normalize with Scikit-Learn by Jeff Hale](https://towardsdatascience.com/scale-standardize-or-normalize-with-scikit-learn-6ccc7d176a02).\n",
        "\n",
        "We've already turned our data into numbers using `get_dummies()`, let's see how we'd normalize it as well."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9v7P20A2d7H6"
      },
      "source": [
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "import tensorflow as tf\n",
        "\n",
        "# Read in the insurance dataset\n",
        "insurance = pd.read_csv(\"https://raw.githubusercontent.com/stedy/Machine-Learning-with-R-datasets/master/insurance.csv\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vir8UAIwlUOo"
      },
      "source": [
        "# Check out the data\n",
        "insurance.head()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SHMQiNosg3J3"
      },
      "source": [
        "Now, just as before, we need to transform the non-numerical columns into numbers and this time we'll also be normalizing the numerical columns with different ranges (to make sure they're all between 0 and 1).\n",
        "\n",
        "To do this, we're going to use a few classes from Scikit-Learn:\n",
        "* [`make_column_transformer`](https://scikit-learn.org/stable/modules/generated/sklearn.compose.make_column_transformer.html) - build a multi-step data preprocessing function for the folllowing trnasformations:\n",
        "  * [`MinMaxScaler`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html) - make sure all numerical columns are normalized (between 0 and 1).\n",
        "  * [`OneHotEncoder`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html) - one hot encode the non-numerical columns.\n",
        "\n",
        "Let's see them in action."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-x9JwbV0hqWh"
      },
      "source": [
        "from sklearn.compose import make_column_transformer\n",
        "from sklearn.preprocessing import MinMaxScaler, OneHotEncoder\n",
        "\n",
        "# Create column transformer (this will help us normalize/preprocess our data)\n",
        "ct = make_column_transformer(\n",
        "    (MinMaxScaler(), [\"age\", \"bmi\", \"children\"]), # get all values between 0 and 1\n",
        "    (OneHotEncoder(handle_unknown=\"ignore\"), [\"sex\", \"smoker\", \"region\"])\n",
        ")\n",
        "\n",
        "# Create X & y\n",
        "X = insurance.drop(\"charges\", axis=1)\n",
        "y = insurance[\"charges\"]\n",
        "\n",
        "# Build our train and test sets (use random state to ensure same split as before)\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
        "\n",
        "# Fit column transformer on the training data only (doing so on test data would result in data leakage)\n",
        "ct.fit(X_train)\n",
        "\n",
        "# Transform training and test data with normalization (MinMaxScalar) and one hot encoding (OneHotEncoder)\n",
        "X_train_normal = ct.transform(X_train)\n",
        "X_test_normal = ct.transform(X_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tz58y3nPiBJ-"
      },
      "source": [
        "Now we've normalized it and one-hot encoding it, what does our data look like now?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VODt2YiziK45"
      },
      "source": [
        "# Non-normalized and non-one-hot encoded data example\n",
        "X_train.loc[0]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mMYDXdwUnNVt"
      },
      "source": [
        "# Normalized and one-hot encoded example\n",
        "X_train_normal[0]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9iI4KtfWib44"
      },
      "source": [
        "How about the shapes?"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### ❓ Q12. Check the shapes of X_train_normal and X_train"
      ],
      "metadata": {
        "id": "B02enwpICiNc"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SFmxzqrWntj7"
      },
      "source": [
        "# Notice the normalized/one-hot encoded shape is larger because of the extra columns\n",
        "____, ____ # YOUR CODE HERE"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# q_12.solution()"
      ],
      "metadata": {
        "id": "lpuICrQ1Cqwl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MST951aYijTS"
      },
      "source": [
        "Our data is normalized and numerical, let's model it.\n",
        "\n",
        "We'll use the same model as `insurance_model_2`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TdHnIQqll83Y"
      },
      "source": [
        "# Set random seed\n",
        "tf.random.set_seed(42)\n",
        "\n",
        "# Build the model (3 layers, 100, 10, 1 units)\n",
        "insurance_model_3 = tf.keras.Sequential([\n",
        "  tf.keras.layers.Dense(100),\n",
        "  tf.keras.layers.Dense(10),\n",
        "  tf.keras.layers.Dense(1)\n",
        "])\n",
        "\n",
        "# Compile the model\n",
        "insurance_model_3.compile(loss=tf.keras.losses.mae,\n",
        "                          optimizer=tf.keras.optimizers.Adam(),\n",
        "                          metrics=['mae'])\n",
        "\n",
        "# Fit the model for 200 epochs (same as insurance_model_2)\n",
        "insurance_model_3.fit(X_train_normal, y_train, epochs=200, verbose=0) "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-DjwktO6jW41"
      },
      "source": [
        "Let's evaluate the model on normalized test set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sBcXZu9AnZfP"
      },
      "source": [
        "# Evaluate 3rd model\n",
        "insurance_model_3_loss, insurance_model_3_mae = insurance_model_3.evaluate(X_test_normal, y_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZlHro290jhtX"
      },
      "source": [
        "And finally, let's compare the results from `insurance_model_2` (trained on non-normalized data) and `insurance_model_3` (trained on normalized data)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ybZtnVlNjCJO"
      },
      "source": [
        "# Compare modelling results from non-normalized data and normalized data\n",
        "insurance_model_2_mae, insurance_model_3_mae"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gUttViY4jzi8"
      },
      "source": [
        "From this we can see normalizing the data results in 10% less error using the same model than not normalizing the data.\n",
        "\n",
        "This is **one of the main benefits of normalization: faster convergence time** (a fancy way of saying, your model gets to better results faster).\n",
        "\n",
        "`insurance_model_2` may have eventually achieved the same results as `insurance_model_3` if we left it training for longer. \n",
        "\n",
        "Also, the results may change if we were to alter the architectures of the models, e.g. more hidden units per layer or more layers.\n",
        "\n",
        "But since our main goal as neural network practioners is to decrease the time between experiments, anything that helps us get better results sooner is a plus."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NhIPO_KqocHP"
      },
      "source": [
        "## 🛠 Exercises\n",
        "\n",
        "We've a covered a whole lot pretty quickly.\n",
        "\n",
        "So now it's time to have a **play around** with a few things and start to build up your initution.\n",
        "\n",
        "I emphasise the words play around because that's very important. Try a few things out, run the code and see what happens.\n",
        "\n",
        "1. Create your own regression dataset (or make the one we created in \"Create data to view and fit\" bigger) and build fit a model to it.\n",
        "2. Try building a neural network with 4 Dense layers and fitting it to your own regression dataset, how does it perform?\n",
        "3. Try and improve the results we got on the insurance dataset, some things you might want to try include:\n",
        "  * Building a larger model (how does one with 4 dense layers go?).\n",
        "  * Increasing the number of units in each layer.\n",
        "  * Lookup the documentation of [Adam](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adam) and find out what the first parameter is, what happens if you increase it by 10x?\n",
        "  * What happens if you train for longer (say 300 epochs instead of 200)? \n",
        "4. Import the [Boston pricing dataset](https://www.tensorflow.org/api_docs/python/tf/keras/datasets/boston_housing/load_data) from TensorFlow [`tf.keras.datasets`](https://www.tensorflow.org/api_docs/python/tf/keras/datasets) and model it.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AyiHG2nubmu7"
      },
      "source": [
        "## 📖 Extra curriculum\n",
        "\n",
        "If you're looking for extra materials relating to this notebook, I'd check out the following:\n",
        "\n",
        "* [MIT introduction deep learning lecture 1](https://youtu.be/njKP3FqW3Sk) - gives a great overview of what's happening behind all of the code we're running.\n",
        "* Reading: 1-hour of [Chapter 1 of Neural Networks and Deep Learning](http://neuralnetworksanddeeplearning.com/chap1.html) by Michael Nielson - a great in-depth and hands-on example of the intuition behind neural networks.\n",
        "\n",
        "To practice your regression modelling with TensorFlow, I'd also encourage you to look through [Lion Bridge's collection of datasets](https://lionbridge.ai/datasets/) or [Kaggle's datasets](https://www.kaggle.com/data), find a regression dataset which sparks your interest and try to model."
      ]
    }
  ]
}